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The article recounts in detail the process used to include all department faculty members in the design of 
outcomes assessment measures for the management major, basing these on broad learning outcomes which 
the department had previously identified. While other literature has described the outcomes assessment 
design process in broad terms, the current article relates specific steps used to produce test questions. The 
article incorporates theory from the outcomes assessment, HRD, and the organizational behavior 
literatures. 
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Assessment of student learning outcomes has become increasingly important since the passage of the No Child Left 
Behind Act. Like grade schools and high schools, colleges and universities are devoting financial and human 
resources to this task. Research shows that higher education faculty members spend the majority of their time 
teaching (Peterson & Wiesenberg, 2006), so it is important to know whether students are learning what is taught. At 
the higher education level, many publications which guide the outcomes assessment process include charts, grids, 
grading rubrics, goals, etc., - each full of words - but these publications do not describe the specifics of how these 
words were derived. 

For example, the set of books. Assessment of Student Learning in Business Schools (Martell & Calderon, 2005) 
contains many articles on the outcomes assessment design process. In the set, one article gives some procedural 
details, but it primarily describes a top-down. Dean and curriculum-committee driven process, and does not state 
specifically how all faculty members were involved (Anderson-Fletcher, 2005). Indeed, most outcomes assessment 
articles discuss theory and generalities but lack discussion of specific practices. 

In the field of human resource development (HRD), a recurring theme is the need to bridge the gap between 
research and practice (Torraco, 2005). The primary goal of the current paper is to bridge this gap by documenting 
the outcomes assessment creation process that was designed to overcome two major obstacles. First, there is a lack 
of published guidelines on the specific steps to take to create outcomes assessment measures. The current paper 
documents the steps that were taken to design the outcomes assessment measures, and these steps can be used by 
other researchers and by practioners when creating training evaluation instruments. Second, in the academic 
department included in the study, there has historically been a very low level of participation in decision making by 
some faculty members. This low level of participation may have been due to their belief that their input had been 
dismissed as unimportant in prior department projects. The current paper documents the steps that were taken to 
ensure that all faculty members were given the opportunity both to participate and also to see that all of their input 
was included in the final outcomes assessment measures. 

In sum, the goal of the current paper is bridge that gap from research to practice by documenting a theory-based 
set of practices that can be used both to create outcomes assessment measures and also to include all department 
members in the decision making process. 

Research shows that organizations perform better when employees are encouraged to participate in decision 
making (Goldstein, 2005). We used a critical reflection process (van Woerkom, 2004) so that the content of each 
step of the process was derived from the content of the previous step. In each subsequent step, each faculty member 
saw how his or her input was included in the assessment project. No input from any faculty member was discarded, 
which helped maintain a high level of participation of the faculty (van Woerkom, 2004). Studies show that the 
process for developing evaluations is equally as important to participants as the type of evaluation developed 
(Sloman, 2004). The process of developing evaluations helps professors rethink what they do in their courses and 
why (Sloman, 2004). 

The process we used was based on the outcomes assessment, organizational behavior, and HRD literatures. 
The outcomes assessment literature was used as the foundation for creating measures in the Management major. 
The organization behavior literature was used as the basis for managing teamwork among faculty members. The 
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HRD literature was used as the basis for writing reliable and valid test questions based on a given body of 
knowledge. 

In the following sections, we begin with a description of the College. Next, we discuss the outcomes 
assessment and related literatures. These are the foundation for the third section, which is a description of the steps 
used to write the instruments. 

Overview of the College 

The College is a public institution, founded in the mid 1800’s, which enrolls approximately 9,100 students. Within 
the College, the School of Management has 29 faculty members and more than 1,100 students enrolled in its six 
majors, within three academic departments: Management and Marketing; Accounting and Information Systems; and 
Finance and Economics. The Department of Management and Marketing offers two degrees: Bachelor of Science in 
Management and Bachelor of Science in Marketing. Within the Management major, the department offers four 
concentrations: general management; operations; human resources; and international. The Management major is a 
junior and senior level program - students may not enroll in the Introduction to Management course (i.e., the 
gateway course) until they have attained junior level status. In January, 2006, when writing the outcomes assessment 
instruments for the Management major began, the department had 1 1 faculty members and 625 students enrolled in 
its two majors (approximately 70% of these were enrolled in the Management major). 

Impetus for Conducting Outcomes Assessment 

As a public institution. The College operates under the auspices of the State Board, which is requiring the public 
colleges in the state to conduct outcomes assessment of student learning. Each academic department has autonomy 
to set learning outcomes for its majors and to design its outcomes assessment measures. The College supports the 
development of the outcomes assessment process and measures by giving each academic department the equivalent 
of one course of release time (3 credits) to be distributed to one or more faculty members, as the department deems 
appropriate. The College also appointed one of its full professors to the position of Assistant Vice President for 
Academic Affairs to be an internal consultant in guiding outcomes assessment development. In addition, the 
College hired an external consultant as a subject matter expert in outcomes assessment to provide guidance to 
departments. The College would like each academic department to report data on student learning outcomes by July, 
2007. Therefore, time was of the essence given that the department had only 18 months to write the measures and to 
gather and report data. 

Overview of Development of Broad Learning Outcomes 

In October, 2004, the members of the Department of Management and Marketing participated in a School of 
Management off-site retreat, during which department faculty members developed broad categories of learning 
outcomes for the Management major - knowledge areas in which department members agreed students should be 
proficient upon graduation (e.g., forecasting, ethics). A brainstorming process was used (Thompson, 2003), within 
which all faculty members of the department gave suggestions regarding potential topics. Six broad areas were 
identified: (1) quantitative analysis and decision making; (2) production and operations management; (3) 
management/interpersonal; (4) legal framework of management; (5) financial analysis and control; and (6) strategic 
management. Sub-topics were suggested for each of these six broad areas, and this list of sub-topics served as the 
basis of the outcomes assessment measures. Although the scope of this list may appear to be broad, it was designed 
to include content from each of the four management concentrations mentioned earlier (i.e., general, operations, 
human resources, and international). 

Theory Supporting the Instrument Development Proeess 

The department designated one faculty member to be the point person to develop the outcomes assessment measures 
for the Management major. (The measures for the Marketing major will be developed by a different point person.) 
The point person’s task was to translate the 86 broad learning outcomes into measures that will be administered to 
the students. 

Using Pre-Packaged Tests Versus Creating Program Specific Tests 

Many pre-packaged instruments are available for measuring student learning in business school programs 
(AACSB, 2006). We chose, however, to write our own measures rather than to use pre-packaged measures from 
organizations such as Educational Testing Service (ETS) for four reasons. 




Measuring program-specific constructs. First, we wanted to measure the body of knowledge we had decided is 
important for our students’ post-graduation success based on our 86 learning outcomes. Pre-packaged assessments 
might lack generalizability across institutions, meaning they might not have equal validity and reliability across 
populations and curriculum (Lohman, 2004). Using pre-packaged tests would tell us whether our students learned 
something, but it might not tell us whether our students learned what we taught (Rotondo, 2005). It is, however, 
critical for instruments to have validity to measure the relevant constructs (Carter, 2001). 

Avoiding teaching to a test. Second, we want to be able to use the data gathered from outcomes assessment to 
improve our curriculum to enhance our students’ learning. We do not want to use data to alter our program so that 
we “teach to a test,” which could be the case if we chose a pre-packaged test. Research has shown that linking 
curriculum to evaluation helps organizations meet their strategic goals by focusing attention on course objectives 
(Allen, 2004; Kirkpatrick & Hawk, 2006). 

Maintaining focus on our college mission. Third, we are willing to sacrifice the ability to benchmark our 
outcomes against the outcomes of other schools (i.e., external validity) in order to more closely measure what is 
taught in our courses (i.e., internal validity) and to maintain the ability to tailor our program to meet our College 
mission. Using pre-packaged tests would essentially allow an outside organization to determine what content areas 
should be emphasized in our curriculum without regard to our College mission. 

Outcomes assessment can be viewed as being a form of training evaluation, which seeks to determine whether 
training goals were met (Alvarez, Salas, & Garofano, 2004). We assume that content areas are taught, and seek only 
to know whether our students learned and retained that content. Class sizes are limited to 30 students, and faculty 
members are expected to have one-on-one interactions with students and to know which pedagogies are most 
effective to increase students’ knowledge, skills, and abilities. The results of outcomes assessment, in conjunction 
with our knowledge of our students, can guide individual faculty members in making changes to their courses if they 
wish to do so. 

Meeting accreditation standards. Fourth, The College is accredited by a regional accrediting agency, which 
does not require use of a pre-packaged test (Mundhenk, 2005). Further the School of Management may seek 
American Association of Colleges and Schools of Business (AACSB) accreditation which also does not require use 
of a pre-packaged test, but which does require colleges to involve all faculty members in the outcomes assessment 
process (Anderson-Fletcher, 2005). 

In the past, accrediting agencies focused on measuring educational inputs, such as budgets, number of faculty, 
course content, etc. (Henniger, 1994). Today, however, there is a greater push to measure outputs - that is, student 
learning. In conducting outcomes assessment, our goal is to measure student performance, not faculty performance. 
This is but one of many possible uses of assessment data from which we could choose (Bober & Bartlett, 2004). It 
is, however, an important goal because students actively and frequently seek feedback on their performance 
(Kuchinke, 2001). In addition, student learning is the primary goal of education, and many business programs focus 
on its measurement within the outcomes assessment process (Apostolou, 1999; Cherry & Dave, 1997; Kretovics, 
1999). 

Potential Difficulties in Gaining Consensus 

The department chose to designate a single point person in order to expedite the process and to diminish the 
impact of four potential difficulties in gaining consensus of department members. 

The need for outcomes assessment. First, in many academic departments, some faculty members 
wholeheartedly embrace the need to conduct outcomes assessment while others are opposed (Martell, 2005). The 
department is mandated to conduct outcomes assessment by the State Board, and using a single point person could 
ensure that the project was completed even if there had been the type of opposition within the department which 
could derail the functioning of a committee. 

Development methods. The second potential difficulty is, among those faculty members who embrace the need 
to conduct outcomes assessment, some will prefer to use one method to create the measures (e.g., brainstorming) 
while other faculty members will prefer to use an alternate development method (Camacho & Paulus, 1995). Using 
a single point person could ensure that all department members help to design the outcomes assessment measures 
while avoiding the potential difficulty that large amounts of time would be used to design the outcomes assessment 
creation process. As mentioned, we used a critical reflection process. This was conducted in a method similar to 
the Delphi technique (Van de Ven & Delbecq, 1974). Members were given written questions and asked to respond 
in writing. These responses were compiled and resubmitted to members. A meeting of the entire group was held 
only at the end of the process. 

Types of outcomes. The third potential difficulty is faculty members at many colleges often disagree about the 
type of outcome to be measured, such as facts versus skills (Rotondo, 2005). For example, one college struggled 
with this issue when creating measures of interpersonal skills (Bommer, Rubin, & Bartels, 2005). Specifically, they 




wondered whether assessment should measure the facts related to interpersonal skills or the use of interpersonal 
skills. Another college chose to measure both knowledge and skills in its marketing program (Davis, Misra, & Van 
Auken, 2002), 

In employee training, we label these outcomes as knowledge (i.e., facts) and behavior (i.e., applying the facts to 
behavior on the job), according to Kirkpatrick’s (1998) model of training evaluation. The United States Department 
of Labor (2006) emphasizes that successful job performance requires employees to possess three characteristics: 
knowledge (i.e., facts and principles); ability (i.e., ability to learn, improve or reason); and skill (i.e., ability to 
perform a task). 

In the test development literature, two well-known taxonomies of learning outcomes were developed by Bloom 
(1956) and Gagne (1985). Bloom’s taxonomy includes knowledge, comprehension, application, analysis, synthesis, 
and evaluation. Gagne’s taxonomy includes intellectual skills, cognitive strategies, verbal information, motor skills, 
and attitudes (Shrock & Coscarelli, 1996). Given that there is a wealth of outcome types from which to choose, 
using a single point person could ensure that all appropriate outcome types are measured by incorporating ideas 
from each department member. 

Type of assessment. The fourth potential difficulty is faculty members often disagree about the type of 
assessment method that should be used (e.g., multiple-choice, case study, oral presentation). Studies show that the 
type of outcome measured should dictate the type of assessment used (Melancon & Williams, 2006). In addition, 
assessments should use multiple assessment methods for results to be reliable and valid (Melancon & Williams, 
2006; Riggio, Mayes, & Schleicher, 2003). Again, using a single point person could ensure that all necessary 
assessment methods are used by incorporating ideas from each department member. 

In summary, although on its face using a single point person seems to imply that the measures were designed 
unilaterally, and could therefore lack comprehensiveness, in the current case, using a single point person ensured 
that each faculty member’s input was given equal weight and that the measures were constructed in a timely manner. 

Specific Steps Used to Create the Instruments 

In this section, we describe the process that was used to meet the two goals of the project: create outcomes 
assessment measures and include all department members in the process. The process is described in detail to 
enable other researchers and practioners to mimic this process when writing their own evaluation instruments. 

The process of translating the 86 sub-topics into tests questions was conducted between February and May, 
2006. The process consisted of ten major steps, most of which invited direct, written input from all department 
faculty members. As stated earlier, the two primary goals of this project were to create outcomes assessment 
measures and to include all department members in the decision making process. Each of the following ten steps 
was conducted to meet either one or both of these goals. The steps began with a literature review, continued with a 
comparison of the six broad learning outcomes and 86 sub-topics to course syllabi, and ended with writing the test 
questions. These steps are based on guidelines for creating outcomes measures in the training evaluation literature 
(e.g., Kirkpatrick, 1998). 

As stated, the College provided release time from teaching to the point person to allow time to conduct the 
project, and appointed an internal consultant and an external consultant to guide the process. The point person met 
with the internal and the external consultants on five occasions from February to May to ensure that the project was 
progressing appropriately. 

Step 1: Literature Review 

The point person reviewed relevant literature from AACSB, ETS, the U.S. Department of Labor, peer-reviewed 
journals, and relevant books regarding outcomes assessment in the management discipline, training evaluation, and 
test development. This was done to ensure our process and our product are aligned with these organizations’ 
standards. A portion of this literature was cited above. This phase of the project was conducted between February 1 
and April 8. 

Step 2: Review of Learning Outcomes 

The point person merged the list of learning outcomes that was produced by department faculty members during 
the October 2004 off-site retreat with learning outcomes lists that were developed during other Management 
department meetings in prior years. This task entailed deleting duplicate items, and inserting remaining items from 
one list into relevant categories on the other list. The goal was to ensure that the list used at the beginning of the 
outcomes assessment project contained all the learning outcomes that had been identified. (It was assumed that the 
final outcomes assessment instrument would measure only a subset of the list, and include only the most critical 
learning outcomes, as per guidance of the internal and external consultants.) 




This resulted in a single list of 86 learning outcomes. The point person used Microsoft Excel to create a grid of 
the 86 learning outcomes (rows) versus the 34 courses required (columns) for the management major. This phase of 
the project was conducted between February 5 and 8. 

Step 3: Review of Course Syllabi 

The point person reviewed Management course syllabi for the 1 8 Management courses that are required for the 
Bachelor of Science in Management to determine which of the 86 learning outcomes are taught in which of the 1 8 
courses. At this time, the list of courses was shortened from 34 to include only the 18 taught in the Management 
department because the department cannot control what is taught in courses offered by other departments, such as 
Math and Accounting. As stated earlier, the department is conducting outcomes assessment only to measure 
whether students learned what was taught in Management courses. 

An “X” was placed into the grid if a topic was covered in a course. In creating a grid, some departments use a 
Likert-type scale ranging from 1 to 3 to indicate the extent to which a topic is covered in a course. In our 
department, it was not possible to determine from syllabi the extent of topic coverage, so the grid was limited to 
indicate only whether a topic was covered at all. This phase of the project was conducted between February 5 and 8. 
Step 4: Request Course Content Detail from Faculty Members 

The point person requested detail on course content from professors whose syllabi lacked detail. For example, 
if a syllabus listed the topic, “ethics,” subtopics could include names of specific theories (e.g.. Principle of the 
Double Effect) and philosophers (e.g., Machiavelli). This information was entered into the grid. Most faculty 
members readily provided detail when shown examples of what was requested. Faculty members were invited to 
participate, but were not required or pressured to do so. Some faculty members chose to refrain from providing 
detail in this stage. This phase of the project was conducted between February 9 and 10. 

Step 5: Request Feedback from Faculty Members on the Match of Sub-Topics to Course Content 

The point person distributed copies of the grid to faculty members and requested their feedback regarding its 
accuracy in matching content areas to specific courses. Nearly all faculty members returned their copies of the grid, 
and included additional “X’s” or, in rare cases, asked that an “X” be deleted from the grid. The point person revised 
the grid as requested, and resubmitted it to faculty for feedback. This phase of the project was conducted between 
February 10 and April 1. 



Table 1. Grid of Outcomes Assessment Content Areas and Required Management Courses 



Core content in Management Area 



Courses and Concentrations within the Bachelor of Science Management Decree 

All Concentrations Gen Gen -HR HR ^ Internal. Operations 



249 301 341 348 461 329 320 322 423 424 425 428 342 345 335 347 355 455 



1 . Quant, analysis & decision making 

(a) Statistical presentations x 

2. Production and operations management 

(a) Forecasting x 

3. Management -interpersonal 

(a) Strategy formulation x 

(b) Managerial decision making x 

(c) Org. goal setting & planning x 

(d) Leadership x 

(e) Communication process in org’s x 

(f) Definition and importance of mgt x 

(g) Managing in a global environment x 

(h) Managing teamwork & group dyn. x 

(i) History of management thought x 

4. Legal framework of management 

(a) Ethics x 

5. Financial analysis and control 
(a) None 

6. Strategic management 

(a) Strategic management model x 

(b) Strategic mgt in global business env. ^ 



Note. The numbers correspond to course numbers for management courses. The course names are as follows: 249 Quantitative business analysis 
II; 301 Introduction to management; 341 Business, government, and society; 348 Operations management; 461 Management seminar; 329 
Organizational theory; 320 Human resource management; 322 Organizational behavior; 423 Compensation and benefits; 424 Labor relations; 
425 Recruitment and selection; 428 Training and development; 342 Comparative management; 345 International business; 335 Process 
analysis; 347 Supply chain management; 355 Quality assurance; and 455 Strategic operations management. 



Step 6: Condense List of Sub-Topics 




The point person condensed the list of sub-topics in the grid. The list was reduced from 86 to 14 sub-topics 
based upon the criterion that a topic should appear in at least three required courses. This is to ensure that a student 
is exposed to the topic multiple times to increase long-term knowledge retention. The topic could be taught in three 
of the five courses that all management majors must take. Alternately, the topic could be taught in two courses that 
all management majors must take and at least one course in each of the four management concentrations (i.e., 
general, human resources, operations, and international). The grid is shown in Table 1. This phase of the project 
was conducted between April 5 and 6. 

Step 7: Generate List of Facts for Sub-Topics 

The point person generated a list of facts for each of the 14 sub-topics based upon course syllabi, the point 
person’s own course notes, and relevant textbooks. For example, under the sub-topic “goal setting and planning” 
facts included “Gantt charts” and “PERT charts.” This phase of the project was conducted between April 7 and 10. 
Step 8: Request Feedback from Faculty Members on List of Facts 

The point person distributed copies of the condensed grid in Table 1 and a separate list of the 14 sub-topics and 
facts for each sub-topic to department faculty members. Faculty members were asked to add and delete facts from 
the list. Most faculty members returned their copies of the list with additions. 

Faculty members who did not return a list after several days were sent an email with the statement, “If you 
would like your input to be included, please send your list to me.” This statement let faculty members know that 
their participation was strictly voluntary, that the project would continue whether or not they participated, and that 
their ideas would definitely be included if they did participate. Several professors sent in their lists following this 
request. This phase of the project was conducted between April 10 and April 15. 

Step 9: Write Test Questions for Each Fact 

The point person created 14 tests, one for each sub-topic. Each test contains at least one question for each fact 
that faculty members identified. The tests are relatively short, approximately one page each, and contain 5 to 20 
questions each. An example of test questions is shown in Table 2. 

The questions were taken from the point person’s own final exams and midterms when possible, or written 
based on the facts identified and based on informal conversations with faculty members that had occurred during 
earlier phases of the project. For example, throughout the project, several faculty members requested that we 
measure knowledge, skills, and abilities, so questions were written to measure each type of learning outcome. Table 
2 contains a sample of questions regarding groups and group decision making that measure knowledge (facts) and 
abilities (thought processes related to groups). This phase of the project was conducted between April 15 and May 9. 

Table 2. Sample of Outcomes Assessment Questions for Group Decision Making 

1 . Give four reasons why people join groups. 

2. Name and define four of the five stages of group development. 

3. What is an advantage of group decision making? 

4. What is a disadvantage of group decision making? 

5. When is better to use a group with 10 members rather than a group with 3 members? 

6. If a group has high cohesiveness, and its goals are not aligned with organization goals, 
what is the effect on group productivity and why? 

Step 10: Request Feedback from Faculty Members on Test Questions 

The point person distributed the revised list of 14 sub-topics and facts to faculty members, and distributed the 
14 tests to faculty members. All faculty members met as a group to discuss the tests. Faculty members who 
expressed concern with the topics of some specific questions or with the high degree of specificity of questions were 
reminded that each question was based on a specific fact that they had identified or approved in a previous step. At 
this meeting, the point person requested that, during the summer, faculty members examine the questions for validity 
and reliability, add and delete test questions, and return the tests in September. This meeting was held on May 10. 

Note that this was the first time the department members met as a group to discuss the project. Between 
February and May, the point person met only informally with individual faculty members to discuss the project. 
Waiting until the test questions were written before holding a group meeting to discuss the test questions ensured 
that each faculty member’s input was given equal weight in the content of the test questions. It also ensured that 
faculty members did not become sidetracked from the task at hand. For example, early in the project, some faculty 
members wanted to design the logistics of outcomes assessment (e.g., types of questions, dates of testing, location of 
testing). The point person was able to make note of these concerns for future use, and to ask that faculty members 
remain focused by saying, “You are on step 9, but I am still on step 3. Let’s do this step first, and then we can figure 




out that step based on what we do in this step.” This statement let faculty members know that their concerns would 
definitely be addressed, but at a later date. 

Remaining Steps 

Although we have developed instruments, they are not finalized. Further, the instruments will need to be pilot 
tested. These steps will occur during the fall 2006 and spring 2007 semesters. We also have not designed testing 
procedures. For example, we could administer one comprehensive test at the end of students’ senior year, and if so, 
administer the test to all students or to a random sample of students. Alternately, we could administer shorter, topic- 
specific tests, and if so, we could embed these into courses or offer them at the end of courses. We also are debating 
whether to administer the instruments so that scores count toward course grades. A body of literature indicates that 
students’ motivation to perform well on tests is low if course grades are not at stake. 

Conclusions 

The current paper documented a process used to bridge the gap between research and practice in writing outcomes 
assessment instruments. We began with a review of the relevant literature (i.e., training evaluation, outcomes 
assessment, organizational behavior, and management). Next, we designed a ten-step process based on this 
literature. This process can be used by other researchers and practioners to design training evaluation instruments. 
Using these steps can be especially beneficial when input from a large group is necessary, when there is opposition 
to the project from a few members, or when some members have not participated in prior projects because they 
believe their input had been treated as unimportant. 

Limitations of the project 

Although the department succeeded in creating outcomes assessment measures, there were several drawbacks to 
using the process described. First, the process was very time consuming for the point person. It is doubtful that a 
faculty member would be able to manage this process without being given release time from other duties. Second, it 
is politically difficult to manage the process. The point person will necessarily be tom between wanting to discuss 
the project and obtain oral input whenever possible, and refraining from allowing randomly-held conversations to 
take precedence over written input. 

Third, members of the department can feel left out, even when providing written input. This is particularly the 
case in a department in which members see each other almost daily and are accustomed to discussing department 
projects. Their inability to discuss the project as a group might lead some members to feel alienated because they 
were unable to have give-and-take with other department members about the project during each of the steps. It is 
noted, however, that in the current project, group discussions were avoided to ensure that no member of the 
department could dismiss another member’s input. 
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